Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
approved these changes
May 12, 2026
ArthurZucker
left a comment
ArthurZucker
left a comment
Collaborator
There was a problem hiding this comment.
LGTM be careful, EP only will fail if you devide by default with tp size the head dim
Comment on lines
166
to
168
| # Account for TP: each KV head is dispatched to a different GPU, so the effective number of KV heads per GPU is | ||
| # simply divided by the TP size (number of GPUs) | ||
| if tp_size is not None and tp_size > 1: |
Collaborator
There was a problem hiding this comment.
only if the attention k and v are target of the tp plan tho
Collaborator
Author
There was a problem hiding this comment.
What could be the other targets? Not familiar enough with the TP plan tbh
Collaborator
Author
There was a problem hiding this comment.
Ok, added a boolean kv_is_tp = "layers.*.self_attn.k_proj" in config.tp_plan and "layers.*.self_attn.v_proj" in config.tp_plan to condition this.
| logit_processor: The [`ContinuousBatchingLogitsProcessorList`] object used to process the logits. | ||
| input_queue: Queue for incoming requests | ||
| input_queue: Queue for incoming requests. Is None if this process is not a TP driver. | ||
| cancel_queue: Queue for cancellation request_ids. Is None if this process is not a TP driver. |
Collaborator
There was a problem hiding this comment.
Okay will read the rest to see if all are drivers or not
jp1924
pushed a commit
to jp1924/transformers
that referenced
this pull request
May 18, 2026
* TP heads and DP / TP seeds * Reproducible hash * Add the notion of TP drivers * Fix NCCL device * Temporary fix for multiple streams * Better handling of NCCL graph mixing * Fix cfg * nit * Move the seed setting * Reworked overall to have accuracy scoring * Adding tests 1/n * Added tests * Style * Fixes * CC review * Nits * Renames * Small fixes * Move distributed stuff to a distributed file * Docstring * Final fixes * Review compliance * Review compliance 2 * Rebase fix * Style * Less redudant testing suite * Fix TP plan * Fix stopping condition * Nits
yuchenxie4645
pushed a commit
to yuchenxie4645/transformers
that referenced
this pull request
May 28, 2026
* TP heads and DP / TP seeds * Reproducible hash * Add the notion of TP drivers * Fix NCCL device * Temporary fix for multiple streams * Better handling of NCCL graph mixing * Fix cfg * nit * Move the seed setting * Reworked overall to have accuracy scoring * Adding tests 1/n * Added tests * Style * Fixes * CC review * Nits * Renames * Small fixes * Move distributed stuff to a distributed file * Docstring * Final fixes * Review compliance * Review compliance 2 * Rebase fix * Style * Less redudant testing suite * Fix TP plan * Fix stopping condition * Nits
kashif
pushed a commit
to kashif/transformers
that referenced
this pull request
Jun 1, 2026
* TP heads and DP / TP seeds * Reproducible hash * Add the notion of TP drivers * Fix NCCL device * Temporary fix for multiple streams * Better handling of NCCL graph mixing * Fix cfg * nit * Move the seed setting * Reworked overall to have accuracy scoring * Adding tests 1/n * Added tests * Style * Fixes * CC review * Nits * Renames * Small fixes * Move distributed stuff to a distributed file * Docstring * Final fixes * Review compliance * Review compliance 2 * Rebase fix * Style * Less redudant testing suite * Fix TP plan * Fix stopping condition * Nits
khushali9
pushed a commit
to khushali9/transformers
that referenced
this pull request
Jun 8, 2026
* TP heads and DP / TP seeds * Reproducible hash * Add the notion of TP drivers * Fix NCCL device * Temporary fix for multiple streams * Better handling of NCCL graph mixing * Fix cfg * nit * Move the seed setting * Reworked overall to have accuracy scoring * Adding tests 1/n * Added tests * Style * Fixes * CC review * Nits * Renames * Small fixes * Move distributed stuff to a distributed file * Docstring * Final fixes * Review compliance * Review compliance 2 * Rebase fix * Style * Less redudant testing suite * Fix TP plan * Fix stopping condition * Nits
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds support for TP in continuous batching. The major changes required to do this were:
hashwhich is salted depending on the processIt also adds a mechanism to the benchmark script to make sure the generation is coherent.
Performance
No perf regression, TP is faster.
Tests
Added tests for TP, all tests run.